test/pytest: a broad raises must name a SQLSTATE (#432) by OffgridwithJD · Pull Request #927 · commandprompt/pgcolumnar

OffgridwithJD · 2026-09-10T16:30:20Z

pytest.raises(psycopg.Error) asserts that one of 254 SQLSTATEs arrived, across 42 SQLSTATE classes — counted against psycopg 3.3.5. It does not reliably assert even that much. Measured on this tree before the guard, this reported 1 passed, exit 0:

with pytest.raises(psycopg.Error):
    conn = psycopg.connect("host=/nonexistent-socket-dir dbname=pgc")
    conn.execute("SELECT pgc_definitely_no_such_function()")
expect.num(1, 1, "the server rejected the call")

What satisfied it was OperationalError with sqlstate None: the connect failed, nothing reached a server, and the statement under test never executed. The test passes, and it passes equally well with the feature it names deleted.

What the layer now refuses

Two shapes, at collection time, found by walking the ast — so an offending file does not collect at all rather than failing one test:

a raises over Error, DatabaseError, Exception or BaseException must bind the exception and pin its SQLSTATE;
any raises block must hold exactly one top-level statement.

expect.sqlstate(exc.value, "42883", name) is the honest form the refusal points at. It compares a typed field, and it refuses three ways of getting out of the claim: an empty set of codes (so the tuple escape hatch cannot become the hole), a two-character SQLSTATE class like "42" (a prefix claim wearing the shape of an exact one), and an exception whose sqlstate is None (which would otherwise compare None against a real code for ever).

AST, not a regex, and the gap is measured rather than asserted: the AST finds 5 pytest.raises call sites in the corpus; a pytest.raises( line regex matches 35 lines. A regex cannot tell code from a string literal, and this file is full of string literals naming the shape it forbids.

The family list is bound inside the scan, not at module level. Every conftest.py under test/pytest/ is imported before collection, so a module-level tuple is writable from the corpus the rule polices: import pgc_vacuity then assign an empty tuple, and the scan reports zero offences for ever with the suite green and nothing saying so. An arm writes three spellings of the name onto the module and requires the refusal to still arrive.

_walk_own deliberately does not descend into a nested def or lambda, because a nested function is its own scope and ast.walk would attribute its with to the enclosing test.

What it does NOT close, stated rather than implied

This closes raises-too-broad, which moves to VACUITY_MODES.md section 2. It only narrows raises-catches-setup, which stays in section 3.4.

The statement rule counts top-level statements, so two shapes still walk past it — each being one statement that performs the setup inside the block: a call to a helper, and a compound statement such as a for holding both the setup and the statement under test. Both are measured at 1 passed, exit 0, zero offences, and both have an arm asserting the scan reports nothing on them, so the residual is a measurement that will be noticed if it changes rather than a gap somebody may later discover.

Evidence

test/selftest/440-a-raises-must-name-a-sqlstate.sh is 55 checks and touches no database. Its fixtures are the ways this guard could quietly stop working, each asserted to be caught: a dropped tuple read, an unwired scan, a drifted offence string (from both the filter side and the producer side), a producer split across two lines, a regex-based scan, a SQLSTATE helper with no five-character check, a test the document does not cover, a suite missing the helper blind-spot arm, and a document that stopped calling the mode open.

On the merged tree (main at f0f1f40 merged in, tree clean, /usr/local/pg17a, under the lock):

harness_selftest.sh: PASSED
checks run: 593
accounting: 593 passed + 0 failed + 0 unrunnable = 593
HARNESS_SELFTEST_RC=0

pytest: 184 passed in 9.17s
PYTEST_RC=0

shellcheck -S error -s bash test/*.sh test/selftest/*.sh is clean, which is CI's exact invocation.

The merge commit

main and this branch both added a numbered TESTS.md section. The merge keeps main's 14 (test_suite_accounting.py) where it is and renumbers this branch's section 17 -> 18, keeping its position at the end of the document, so no prose moves and nothing is dropped.

Verified structurally rather than by eye: 18 headings, 18 TOC entries, numbers contiguous 1..18, titles identical between the two lists, and every TOC anchor equal to the anchor GitHub derives from its heading.

Relationship to the open stack

No collision: this adds selftest part 440, while #923 adds 400 and #925 adds 410. It will conflict with both in TESTS.md and CHANGELOG.md — the append-at-the-top kind, not a code conflict — and I will resolve it whichever way round they land.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

`pytest.raises(psycopg.Error)` asserts that one of 254 SQLSTATEs arrived. Measured against psycopg 3.3.5, by walking its own exception classes: Error, DatabaseError 254 SQLSTATEs across 42 classes OperationalError 88 across 15 DataError 68 across 1 ProgrammingError 57 across 10 InternalError 20 across 5 IntegrityError 7 across 1 NotSupportedError 1 across 1 Warning, InterfaceError 0 So a test can assert "the server rejected it" while the server was never reached. On unmodified main the offending test is `2 passed`, exit 0, and the error that satisfied the raises was an `OperationalError` whose `sqlstate` is None -- a connection that never opened. Against a live PostgreSQL 18.4 the same shape is worse than vacuous: the setup line raises 42602 while the statement under test raises 42704, so the test passes on an error it was not written about. `expect.sqlstate(exc.value, "42704", name)` is the honest form, and a collection scan refuses the broad families that name no SQLSTATE unless the block pins one. WHY Warning AND InterfaceError ARE NOT REFUSED: zero SQLSTATEs each, measured above. Demanding one of them would be a guard nobody could satisfy, and an unsatisfiable guard is how a guard gets switched off. The four intermediate families are not refused either -- narrowing to `OperationalError` is already a claim about the error, and it legitimately covers failures with no SQLSTATE at all. `raises-catches-setup` IS NOT CLOSED, ONLY MITIGATED, and it stays in section 3.4 rather than moving to section 2. Measured on this guard: a helper called from inside the block is one top-level statement and can raise from its own setup, so with a narrow class and a pinned SQLSTATE the offending shape is `1 passed`, exit 0, and the scan reports zero offences. A single compound statement -- a `for` holding several executes -- walks past the one-statement rule the same way. Naming the unclosed id in backticks inside section 2 made the document's own counter report 27 refused against 26 stated, which is the arbiter catching the attempt. THE STATIC BUDGET, counted over the pre-existing corpus: five `pytest.raises` call sites, four of them `pytest.raises(RuntimeError)` in test_build_refusal.py asserting on the message and the fifth `pytest.raises(VacuityError)` at test_guards_pinned.py:169. The scan reports 0 offences on all five. A naive grep over the tree counts 35 occurrences, and the difference is the point: the rest are inside `pytester.makepyfile` strings and in prose, which an ast walk does not see and a line regex would have. That distinction is why the layer's broad-`except` refusal had to be rewritten once already. 22 test functions under test/pytest/ 55 static checks in test/selftest/440-a-raises-must-name-a-sqlstate.sh harness_selftest on pg17a 497 passed + 0 failed + 0 unrunnable = 497, PASSED pytest corpus 170 passed RESIDUAL, beyond the unclosed sibling: the scan reads `with` blocks inside function bodies, so a module-level raises or the plain-call form `pytest.raises(E, fn, arg)` is invisible; the SQLSTATE pin is matched by NAME rather than by dataflow, so a pin inside a helper is not followed; `match=` is deliberately not accepted as a pin, being a regex over the message, which is the substring claim this layer exists to remove; and only files carrying collected test items are scanned, so a helper module is not. AND ONE THAT IS NOT THIS BRANCH'S TO FIX: `_BROAD_RAISES` is a module-level name, so a conftest rebinding it to `()` turns the first rule off. That is true of every module-level name the plugin keeps, `_ORDER_KILLERS` in main included, and it is filed as commandprompt#924 with a reproduction against main rather than patched here. EVERY PREMISE NAME IN THIS PART NAMES ITS OWN SUBJECT, which is not a style preference. The boilerplate premise names say "this part" so they can be copied into any part, and main already carries two copies of `premise: the pytest layer is where this part thinks it is` (in parts 360 and 370). commandprompt#918's ledger keys a check's history on its NAME, so every sharer is one row and one of them going red marks them all as observed red -- a claim about a check nothing attacked. This part's premises therefore say the broad-raises part instead, and it contributes zero duplicated names to test/selftest/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

Both sides added a numbered TESTS.md section. main's 14 (test_suite_accounting.py) is kept where it is; this branch's section keeps its position at the end of the document and is renumbered 17 -> 18, so no prose moves and nothing is dropped. Verified structurally rather than by eye: 18 headings, 18 TOC entries, numbers contiguous 1..18, titles identical between the two lists, and every TOC anchor equal to the anchor GitHub derives from its heading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

jdatcmd

Reviewed at 50c2f5e1. I ran the scanner rather than reading it: a worktree of the branch, pgc_vacuity imported standalone against pytest 9.1.1 with no psycopg, and selftest 440 driven under a stub check. Baseline reproduces your numbers — 440 is 55 checks, 55 passed, 0 failed.

Three findings. Each one contradicts a claim the PR body makes for itself, which is why I am asking for changes rather than filing them as notes.

1. The pin is satisfied by mentioning `.sqlstate`, with nothing asserted (blocking)

The body says a broad raises "must bind the exception and pin its SQLSTATE". _sqlstate_pinned_names counts any ast.Attribute named sqlstate and records its root, so the token appearing anywhere in the function turns the broad-family rule off. It never requires the attribute to reach an assertion.

Measured — six files through _raises_sites, controls first so the instrument is visible:

OFFENCE  A control: unpinned broad raises (MUST be an offence)
clean    B bare `.sqlstate` expression, asserts nothing
clean    C assigned to a variable never read
OFFENCE  E positional, broad, unbound, two statements
clean    F a real pin (MUST be clean)

A and F are the controls: the scan does fire, and does not false-positive on the honest form. B is this:

with pytest.raises(psycopg.Error) as exc:
    conn.execute("SELECT pgc_no_such()")
exc.value.sqlstate                    # bare expression, asserts nothing
expect.num(1, 1, "the server rejected the call")

That is byte-for-byte the vacuity the PR is named for — any of the 254 SQLSTATEs satisfies it — and it collects clean. C is the same with code = exc.value.sqlstate. This is not in your stated residual list, so a reader is told the mode is closed.

2. All three "a neutered arm is caught" fixtures are unfaithful, and the faithful neutering stays green (blocking)

This is the one I would fix first, because the Evidence section rests on it: "its fixtures are the ways this guard could quietly stop working, each asserted to be caught".

Fixture 3 writes if False and broad and (bound is None or b not in p): — it renames bound→b and pinned→p as well as prefixing False and. The arm greps _rq_body for the literal bound not in pinned. So it is the rename that drops the count to 0. The same shape is in fixture 4 (!= 1 rewritten to >= 1) and fixture 5 (the isinstance dropped).

Measured on the real pgc_vacuity.py, one arm at a time, each mutation diffed to prove it applied and the file restored byte-identical after:

faithful mutation	440 says	the guard
`if False and broad and (bound is None or bound not in pinned):`	55 checks, 55 passed, 0 failed	case A goes clean — blind to the unpinned broad raises
`if False and sites and len(node.body) != 1:`	55 checks, 55 passed, 0 failed	case E loses its two-statement offence
`... if isinstance(arg, ast.Tuple) and False else [arg]`	55 checks, 55 passed, 0 failed	`raises((ValueError, psycopg.Error))` goes OFFENCE → clean

Each mutation leaves the pinned substring intact, so a text arm cannot see it. The arms are text pins: 440 never executes the scanner — 0 python3 invocations, and 44 of its 55 checks are grep -c against the function's source text. A text pin catches a rewrite or a deletion. It cannot catch False and, which is how a guard actually dies.

The fix that closes all three at once is a behavioural arm: write a fixture file that contains the vacuity, run the scan over it, and assert the offence count. That arm fails under any neutering, faithful or not.

This matters more than it otherwise would because at this head nothing runs the corpus at all: grep -rn pytest .github/workflows/ returns 0 matches, and SUITES in run_all_versions.sh names no pytest entry. test_raises_sqlstate.py's 22 arms are never executed by the gate, so 440 is the only enforcement — and 440 is green under all three neuterings above.

Worth knowing: #921 is the PR that introduces the job which would run this (10 pytest matches in .github/workflows/ at its head, versus 0 here). That is an argument for landing #921 first, not against this one.

3. The keyword form escapes both rules (should-fix)

if tail != "raises" or not call.args: continue skips the item before it is appended to sites, so pytest.raises(expected_exception=...) is checked by neither rule. Measured — D and E differ only in how the argument is passed:

clean    D  pytest.raises(expected_exception=psycopg.Error):  broad, unbound, two statements
OFFENCE  E  pytest.raises(psycopg.Error):                     broad, unbound, two statements
           -> names no SQLSTATE
           -> the block holds 2 statements, so which one raised is not pinned

The body states rule 2 unconditionally — "any raises block must hold exactly one top-level statement". As written that is false: the statement rule is silently conditional on the class being passed positionally. Either read call.keywords or narrow the sentence and add the form to the residual list.

Aliased imports (from psycopg import Error as PgErr) escape too. That one is inherent to a name-matching AST scan, so I would put it in the residual list rather than in the code.

What I could not fault

The AST-over-regex choice holds up: I re-measured the 35/5/30 and 22/8 splits and they reconcile. expect.sqlstate itself cannot pass vacuously — want must be five [0-9A-Z] characters and must equal got. The function-local family list really is unreachable from the corpus it polices, and the tuple-member hole from #905 is genuinely closed at the unmutated head. _walk_own not descending into a nested def is right, and the reasoning in the comment is right.

The guard is worth having. What it needs is an arm that runs it.

@linuxhikerpm

… an exemption, and the arms live in one harness (commandprompt#432) @linuxhikerpm found three defects and all three were real. Each is reproduced below before it is fixed, and the third one changed the shape of the change rather than a line of it. 1. MENTIONING `.sqlstate` SATISFIED THE PIN. `_sqlstate_pinned_names` counted any `ast.Attribute` named `sqlstate` anywhere in the body and never required it to reach an assertion. Measured through `_raises_sites`, controls first: OFFENCE A unpinned broad raises (the instrument works) clean B `exc.value.sqlstate` as a bare statement <- asserts nothing clean C `code = exc.value.sqlstate`, never read <- asserts nothing clean F a real pin (no false positive) B and C are byte-for-byte the vacuity this guard is named for. The rule now requires the read to reach a CALL, and follows ONE hop of assignment so the honest `code = exc.value.sqlstate` / `expect.text(code, ...)` form is not refused. One hop, not two: it is a floor, and the floor is stated rather than implied. Four honest forms were re-measured as controls and all four stay clean. 2. THE KEYWORD FORM ESCAPED BOTH RULES. `if tail != "raises" or not call.args: continue` skipped the item before it was recorded, so `pytest.raises(expected_exception=E)` was checked by neither. The two forms differ in nothing else: clean D pytest.raises(expected_exception=psycopg.Error) broad, 2 statements OFFENCE E pytest.raises(psycopg.Error) broad, 2 statements So the statement rule was silently conditional on the class being positional while the documentation stated it unconditionally. The class is now read from `args` or from the `expected_exception` keyword, and D reports the same two offences as E. 3. THE NEUTERING FIXTURES WERE UNFAITHFUL, AND THE FIX IS STRUCTURAL. All three renamed a variable AS WELL AS prefixing `False and` -- `bound`->`b`, `pinned`->`p` -- and the arm greps for the literal `bound not in pinned`. So the rename was what dropped the count to 0 and the `False and` was decoration. Measured on the real file, faithfully, nothing renamed: if False and broad and (bound is None or bound not in pinned): 55/55 GREEN, blind if False and sites and len(node.body) != 1: 55/55 GREEN, blind The root cause was that the shell part never ran the scanner: 44 of its 55 checks were `grep -c` against the function's text, 0 `python3` invocations. MY FIRST FIX WAS WRONG. I extracted the scan into a pytest-free module so the SHELL part could drive it -- which closed the coverage hole by creating a dependency. jd's rule, set while I was doing it: the shell tests and the pytest corpus are PARALLEL IN FUNCTIONALITY and must not call, import or reference each other outside docs. Each asserts against the product, in its own terms, never against the other harness's implementation. So `test/selftest/440-a-raises-must-name-a-sqlstate.sh` is DELETED rather than repaired. Its whole subject was this layer's source text, which is the coupling, and its text pins could not see the thing they claimed to. Every property it checked is either already covered behaviourally here or was one of those pins; the one that was not -- that `raises-catches-setup` stays listed as open -- is now an arm in this corpus, reading the document, which is the one place the two harnesses may meet. The neutering proof is now two arms that copy the layer, disable ONE condition with `False and` and nothing else, and require the copy to go blind while still containing the substring a grep arm would have pinned. They fail under any neutering. MEASURED pytest corpus 192 passed (22 -> 30 arms in test_raises_sqlstate.py) harness_selftest 538 checks, 538 passed + 0 failed + 0 unrunnable, rc 0 the six shapes A offence, B offence, C offence, D 2 offences, E 2 offences, F clean honest controls direct field read, one hop, narrow family, keyword+pin -- all clean Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

OffgridwithJD · 2026-09-10T18:04:07Z

@linuxhikerpm all three findings were real, all three are fixed at 294cb8cbca297e7f0efe6d2adf97abff066bceb8, and the third one changed the shape of the change rather than a line of it. I reproduced each before touching anything.

1. Mentioning `.sqlstate` satisfied the pin

Reproduced through _raises_sites, controls first so the instrument is visible:

	case	before	after
A	unpinned broad `raises`	offence	offence
B	`exc.value.sqlstate` as a bare statement	clean	offence
C	`code = exc.value.sqlstate`, never read	clean	offence
F	a real pin	clean	clean

The rule now requires the read to reach a call, and follows one hop of assignment so the honest form is not refused:

code = exc.value.sqlstate
expect.text(code, "42883", "the function does not exist")   # still collectable

One hop, not two — it is a floor and the floor is now stated in the docstring rather than implied. I re-measured four honest forms as controls (helper pin, direct field read, one hop, narrow family) and all four stay clean, because a rule that refuses honest tests is a rule somebody switches off.

2. The keyword form escaped both rules

Your pair reproduced exactly: D clean, E two offences, differing in nothing but how the class is passed. The class is now read from args or from the expected_exception keyword, and D reports the same two offences as E. A pinned keyword form stays collectable, so the fix is not "refuse the keyword form".

You were right that the body's sentence was false as written. It is no longer conditional, so the sentence is now true rather than narrowed.

3. The neutering fixtures — and my first fix for them was wrong

Your measurement is exact, and the part I had not seen is that the False and in my fixtures was decoration: the arm greps for bound not in pinned, and the fixture renamed bound→b and pinned→p, so the rename is what dropped the count to zero. Fixtures 4 and 5 have the same shape. I had written the memo about a presence grep being unable to see a disabled arm before I wrote those fixtures.

My first fix created a worse problem. I extracted the scan into a pytest-free module so the shell part could drive it — which closed the coverage hole by creating a dependency between the two harnesses. jd set the rule while I was doing exactly that: the shell tests and the pytest corpus are parallel in functionality and must not call, import or reference each other outside docs. Each asserts against the product, in its own terms, never against the other harness's implementation. (It is now written down in #932.)

So test/selftest/440-a-raises-must-name-a-sqlstate.sh is deleted, not repaired. Two reasons pointing the same way: its whole subject was this layer's source text, which is the coupling; and its text pins could not see the thing they claimed to. Before deleting it I listed all 55 of its check names against the corpus's arms, so the removal is not a silent loss — every property was either already covered behaviourally here or was one of those pins. The single exception, that raises-catches-setup stays listed as open, is now an arm in this corpus reading the document, which is the one place the two harnesses may meet.

The neutering proof is now two arms that copy the layer, disable one condition with False and and nothing else, import the copy, and require it to go blind while still containing the substring a grep arm would have pinned:

test_disabling_the_sqlstate_rule_makes_the_scan_blind
test_disabling_the_statement_rule_makes_the_scan_blind

They fail under any neutering, faithful or not. And because they run the scan, they also make your point about enforcement land where it should: this corpus is where the guard executes, and #921 is the PR that puts this corpus in CI — which is an argument for landing #921 first, as you said.

Evidence

pytest corpus       192 passed   (test_raises_sqlstate.py: 22 -> 30 arms)
harness_selftest    538 checks, 538 passed + 0 failed + 0 unrunnable, rc 0
the six shapes      A offence · B offence · C offence · D 2 · E 2 · F clean
honest controls     direct field, one hop, narrow family, keyword+pin — all clean

The selftest count dropping from 593 to 538 is the 55 deleted checks, and the run is green without them, which is the check that nothing else leaned on that part.

What I am not claiming

Aliased driver imports (from psycopg import Error as PgErr) still escape, and you were right that this is inherent to a name-matching AST scan rather than a bug in it. It is in the residual list in the docstring rather than in the code. The two raises-catches-setup residuals are unchanged and still pinned by their own arms.

jdatcmd

Approving at 294cb8cb, 12/12 SUCCESS, CLEAN. I re-ran the attacks that produced my three findings rather than reading the diff.

All three are closed, with the controls still holding. My original six probe files through _raises_sites:

OFFENCE  A  control: unpinned broad raises          (was OFFENCE, still is)
OFFENCE  B  bare `.sqlstate` expression             (was clean -- now caught)
OFFENCE  C  assigned to a variable never read       (was clean -- now caught)
OFFENCE  D  keyword form, broad, unbound, 2 stmts   (was clean -- now caught by BOTH rules)
OFFENCE  E  positional control for D                (was OFFENCE, still is, both rules)
clean    F  a real pin                              (was clean, still clean)

F is the one that matters as much as B and C: the fix refuses more without refusing the honest form.

The removal proofs are now real, which was my blocking finding. The exact faithful mutation that left selftest 440 at 55 checks, 55 passed, 0 failed while the guard went blind:

if False and broad and (bound is None or bound not in pinned):   ->  8 failed, 22 passed
if False and sites and len(node.body) != 1:                      ->  3 failed, 27 passed
_sqlstate_pinned_names returning set() unconditionally           ->  6 failed, 24 passed

Baseline 30 passed; each mutation diffed to prove it applied; pgc_vacuity.py restored byte-identical afterwards. Text pins could not see False and. These arms run the scanner, so they can.

Deleting selftest 440 rather than repairing it is the right call under the harness-independence rule, and it closes the finding at the root instead of patching three fixtures.

One sequencing fact, not a fault in this PR. Nothing runs these 30 arms in CI at this head — grep -rn pytest .github/workflows/ returns 0. The arms themselves need no cluster (27 take pytester, 2 tmp_path, 1 only expect), but this branch's conftest.py:15 imports psycopg at module scope, so they cannot run driverless here either. #921 fixes exactly that and adds the job that would run them. So this guard becomes enforced when #921 lands, and not before. Worth saying plainly in the PR body rather than leaving a reader to discover it.

Nothing blocking. Good fix.

@jdatcmd

Both PRs moved a mode from VACUITY_MODES.md section 3 to section 2, so both edited the count rows and both claimed TESTS.md section 18. RESOLVED BY COMPOSING, not by choosing: section 2 keeps BOTH rows, the counts become 27 refused / 45 not refused against an unchanged 72 named, and the prose totals the inventory gates follow -- section 2s opening, section 3s opening, the closing paragraph, TESTS.md and README.md. The raises section keeps 18 and the sentinel section becomes 19. Verified structurally rather than by eye: 19 headings, 19 TOC entries, numbers contiguous 1..19, titles identical between the two lists, every TOC anchor equal to the anchor GitHub derives from its heading. AND THE FILE THAT DID NOT CONFLICT IS THE ONE THAT NEEDED RUNNING. git auto-merged pgc_vacuity.py, composing the raises scanner and the sentinel refusal without either side ever having run the composed file -- each was green only against its own base. @jdatcmd asked for the arms on the MERGED tree rather than on the branch, which is where each-green-separately-broken-together lives: test_raises_sqlstate.py + test_failed_query_sentinel.py 40 passed the whole corpus 203 passed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

…080 sweep fix One conflict, CHANGELOG.md, and both sides append a bullet at the top of the same section -- kept both. commandprompt#927 deleted test/selftest/440 on main and this branch never touched it, so nothing else met. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

…harness-guards branch Two conflicts, both additive. CHANGELOG.md: two regions, both sides appending at the top of the same section -- kept both. TESTS.md: both sides number a section, and the raises section main landed at 18 collides with this branch, so it is renumbered 20 and its TOC entry and anchor follow. Verified structurally: 20 headings, 20 TOC entries, numbers contiguous 1..20, titles identical between the two lists, every TOC anchor equal to the anchor GitHub derives from its heading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

…ies, and the job's list is the intersection (commandprompt#432) commandprompt#927 landing made this branch's own arm fire, by name, which is what it exists to do: disagreements: [1: undeclared:test_raises_sqlstate.py] THE OBVIOUS FIX WAS WRONG, and the arm that caught it was right. Declaring the file turned the driver-free job red: four of its arms fail with psycopg shimmed out, because the modules it hands to `pytester` import the driver. It requests no cluster fixture and it still cannot run where there is no driver. MY SECOND ATTEMPT WAS ALSO WRONG, and an existing arm refused it. Folding the driver condition into `partition()` contradicted `test_the_classifier_is_not_fooled_by_prose_that_names_the_driver`, which asserts that a generated inner test requesting a cluster fixture is the INNER run's requirement and not this file's. That arm is correct, and breaking it was the signal that I was overloading one property with two meanings. SO THERE ARE TWO PROPERTIES, derived separately: partition() -> does this file request a cluster? driver_dependent() -> does it need psycopg IMPORTABLE, even with no cluster? job_runnable() -> the intersection, which is what the job can run and `membership_report` compares the declaration against the intersection, with `needs-the-driver:` as a kind of its own -- `needs-a-cluster:` would be a wrong diagnosis and the reader's next action differs. TWO CONDITIONS FOR THE DRIVER PROPERTY, because a driver import in a string is not enough on its own. `test_harness_deps_classifier.py` writes fixture corpora containing `import psycopg` and only ever PARSES them -- nothing imports those files. Reading the string alone would have thrown that file out of the gate it exists to be in. The difference is whether the file drives `pytester`. cluster-free 9 files, including test_raises_sqlstate.py driver-dependent test_raises_sqlstate.py job runnable 8, which is the declaration disagreements [] Three arms pin it, including both controls: a file that only parses a driver import stays job-runnable, and prose naming the driver is not a dependency on it. MEASURED harness_selftest 550 checks, 550 passed + 0 failed + 0 unrunnable, rc 0 pytest corpus 227 passed the gated set as CI runs it 8 files, 150 passed, psycopg asserted absent Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

@OffgridwithJD

Two corrections from @OffgridwithJD's review, and the first one is the document's own rule 3 catching the document. "39 call sites" was 36 calls plus the 3 definitions. The pattern `[^_a-z]_sh(` matches `def _sh(` as readily as a call, which is the same class of error as `[a-z_]+\.sh` matching `sharedir` -- already written three lines above as the thing not to do. Counted with ast now, and the entry says how, because a number in this section has to be re-derivable or it does not belong here. The heading said 4 python files. Three are on main; the fourth arrives with PR #923. The entry always said so, the heading did not, and a reader who stops at the bold line gets a count that is wrong today. Recounted against main at aa53c1b, after #927 and #931 landed: still 3 python files and 7 shell files. test_raises_sqlstate.py, new on main, adds neither -- it drives pytester, not the shell. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK

#927, #931 and #930 landed while this waited on review, so main gained three CHANGELOG entries and two TESTS.md sections. CHANGELOG: both sides append at the top of the same section and neither replaces anything, so the union is the resolution. TESTS.md: the numbering collided. This branch inserted its file section at 15 and pushed "Adding a test", "What this corpus does NOT yet refuse" and "Traps this corpus records" to 16-18; main kept those at 15-17 and appended its two new file sections as 18 and 19. Auto-merge produced two sections numbered 18. Resolved main's way, because main's convention is now to append a new file section after the tail sections: this branch's section becomes 20, and the three tail sections go back to main's 15, 16 and 17. That renumbers one section of this branch rather than two of main's. Checked rather than eyeballed, because an anchor that stops resolving does not announce itself: 20 headings against 20 TOC entries, every TOC text equal to its heading, every anchor equal to what GitHub derives from that heading, and the numbering contiguous 1..20. On the composed tree: selftest 350 41 checks 0 failed, selftest 400 64 checks 0 failed, selftest 080 15 checks 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK

…pt#930 landed (commandprompt#432) The arm named it rather than leaving a hole: cluster-free, not driver-dependent, so the driver-free job can run it and the declaration has to say so. Third time this arm has caught a merge-order consequence rather than a mistake -- commandprompt#922 brought test_suite_accounting.py, commandprompt#927 brought test_raises_sqlstate.py (which turned out to need the DRIVER and so is correctly excluded), and commandprompt#930 brings this one. declared 9 · cluster-free 10 · driver-dependent 1 · disagreements [] corpus 238 passed · the gated set 9 files, 161 passed with psycopg absent Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

#923 moved five times while this waited, and #921, #927, #930, #931 and #932 landed on main underneath it. Composing found two things that a clean merge would not have. TESTS.md: the base now carries 22 sections, and this branch had inserted the ledger at 16. Auto-merge kept the base's tail sections AND this branch's copies of them, so the file would have had two of each. Resolved by keeping only the ledger section from this side, renumbered to 23, where the base's own 22 already ends. Checked rather than eyeballed: 23 headings against 23 TOC entries, every TOC title equal to its heading, every anchor equal to GitHub's derivation, numbering contiguous 1..23, no duplicate heading. NO_CLUSTER gains test_mutation_ledger.py. #921's classifier arrived on the base and immediately named it: membership_report: [1: undeclared:test_mutation_ledger.py] It drives test/pgc_ledger.py, a python tool rather than the shell harness, so it needs neither a cluster nor psycopg. Measured rather than assumed: 9 passed in a venv with no driver, and driver_dependent() agrees. The derived job goes to 11 files and 179 passed. THE LEDGER IS REGENERATED, AND THAT IS THE POINT OF THIS MERGE RATHER THAN A SIDE EFFECT. #923 added 17 checks to selftest 400 and converted 25 skip sites, none of which the committed ledger had ever seen. The gate refuses a check it has never seen, so the composed tree would have failed CI for a reason with nothing to do with either change. Regenerating is the documented repair, and the budget file says so. From a real run of the composed tree, not a synthesised log: harness_selftest.sh: PASSED, rc=0 checks run: 735 | accounting: 735 passed + 0 failed + 0 unrunnable + 0 skipped ledger: 701 rows -> 734 | never=734, ever red=0 gate: new this run=0 Reconciled: 735 records == 732 distinct (suite, part, name) + 3 names that each appear twice in one run, and 0 log triples are missing from the ledger, which is exactly what the gate refuses. All 734 rows carry five fields and none ends in a tab. The budget's asserted census follows to 734. Two ledger rows do not appear in this log -- selftest 330's "all three runner functions", which #923 changed to five. rename-scan reports appeared=0, vanished=2. THEY ARE LEFT DELIBERATELY: this log is PG17 only, and pruning rows that a single major did not produce would delete checks that legitimately run elsewhere. The gate refuses unseen checks, not unused rows. Gates on the composed tree: 350 53/53, 400 81/81, 410 96 checks 0 failed, 080 15/15, shellcheck rc=0 over the whole harness, driver-free job 11 files 179 passed, membership_report []. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK

@linuxhikerpm

@linuxhikerpm's blocking finding, and they are right on a point I had got wrong. I had listed test_check_results_are_machine_readable.py in CONTEXT.md's debt inventory as "arrives with PR #923". A file that has not landed is not pre-existing debt: the harness-independence rule is already on main, and this PR would have introduced a fresh violation of it. The file sources the real test/lib.sh, executes a fixture that sources it, and extracts pgc_reconcile_records from run_all_versions.sh. Under the owner's ruling -- "each harness is independent and should only parallel test functionality" -- that is the coupling rather than the twin: it agrees with the shell by construction and can never report it wrong. DELETED RATHER THAN REWRITTEN, deliberately. The remedy the review asks for is a pytest-native equivalent, and the pytest harness has no per-assertion record stream to be native to: Expect counts assertions, and user_properties carries only the unrunnable state. So "independently implement identity, verdict, reason, sanitization, SKIP accounting and count reconciliation" is a FEATURE in the pytest layer, not a port of this file. Filed separately rather than grown onto a change that is already large. The precedent is #927, where the shell part whose subject was a python module's source text was deleted rather than repaired. What this PR keeps is the shell work, which is what the shell harness owns: strict six-field RESULT validation with the emitter's own verdict list, tab, CR and newline sanitisation, 25 named SKIP sites routed through pgc_record, the derived anti-drift sweep, and the static arm for a check_skip reading a name its file never assigns. Removed the NO_CLUSTER entry, the TESTS.md section and its TOC entry, and the CONTEXT.md debt line that named a file which will no longer arrive. Checked rather than eyeballed: 21 headings against 21 TOC entries, contiguous 1..21, every anchor equal to GitHub's derivation; membership_report []; the derived job 9 files, 161 passed. selftest 350 53/53, 400 81/81, 080 15/15, shellcheck rc=0. And the property the review turns on, measured rather than asserted: of the python files this branch touches, none now reaches into shell. test_suite_accounting.py still does, but it landed on main in #922, it is named in the CONTEXT.md inventory, and this branch's seven changed lines there add zero shell references. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK

`raises-catches-setup`, entry 5 and the last on VACUITY_MODES.md's list of what to add next. The statement COUNT rule refused a `pytest.raises` block holding more than one top-level statement. Two shapes are ONE statement and still perform the setup inside the block, so the count saw nothing: with pytest.raises(psycopg.errors.UndefinedObject) as exc: _setup_then_run(conn) # a HELPER CALL: one statement with pytest.raises(psycopg.errors.UndefinedObject) as exc: for stmt in (setup_sql, sql_under_test): # a COMPOUND STATEMENT: one conn.execute(stmt) # statement holding two Both were measured against the shipped scan reporting `1 passed`, exit 0, zero offences, with the setup raising and the statement under test never running. The two arms that recorded them as residuals now assert the refusal, so the closure is a measurement rather than a sentence. THE FIX IS NOT A DEEPER COUNT, for the reason the entry always gave: counting recursively would also refuse a legitimate single-statement loop. It is a claim about WHICH statement raised, in two rules. NO COMPOUND STATEMENT, and all nine kinds rather than the one the inventory named. The document named the `for` spelling; `if`, `while`, `with` and `try` nest identically, and a rule catching only `for` would close an example rather than a mode. The kinds are looked up by name instead of written out, because `TryStar` and `Match` exist only on newer Pythons and a missing attribute would be a NameError at import rather than a rule that quietly does less. NO CALL TO A FUNCTION DEFINED IN THE SAME FILE, anywhere in the statement. Such a function can run any number of statements and nothing in the block says which failed. A call to an IMPORTED function, or to a METHOD, is the thing under test and stays allowed. THE RULE TURNS ON WHERE THE FUNCTION IS DEFINED, NOT ON THE STATEMENT BEING A CALL, and that is what makes the budget zero. Measured over the corpus before writing either rule: five `pytest.raises` blocks in real code, NONE touching a database -- four call `build_and_install`, imported from the module under test, and one calls a method. The scan reports no offence on any of them. MEASURED FIRST, AND IT CHANGED THE DESIGN. The entry suggests "a helper that runs exactly one statement and owns the assertion", which would be a DB helper. There are zero SQL-raising `pytest.raises` blocks in the corpus, so that helper would have had no call sites -- an instrument with nothing exercising it, which is the thing this directory refuses to build. The shape rules close the same mode against the code that exists. THE RESIDUAL IS A METHOD. A method that performs setup and then the statement is invisible to this rule, and no static rule can see inside it. Stated rather than hidden, and pinned by the arm that accepts the method shape. Prove by removal, five mutations, each by exact string with a parse assertion, and the restore verified byte-identical rather than by `git diff`: control 34 passed compound rule removed 2 failed local-def rule removed 2 failed only `for` counts as compound 1 failed only a bare call is searched 1 failed the message filter narrowed 4 failed TWO OF MY OWN ARMS COULD NOT FAIL UNTIL I MUTATED THEM. The local-def rule searches the WHOLE statement, and my first arm used a bare call only -- so narrowing the search to `Expr` changed nothing and `x = _helper()` stayed open. And the every-compound-kind arm exists because without it, reducing the kind list to `For` alone was invisible. AND THE RULES FIRED WHILE PRINTING NOTHING. The message assembly buckets offenders by substring, and the bucket's filter was the exact sentence of the COUNT rule -- so both new rules refused the run and the layer said `refuses this run: .` with an empty list. The arms reddened on a missing message while the refusal itself worked, which is a guard that cannot be told from an unfired one. The filter is now the common tail of all three phrases. RECORDED WHILE I WAS IN THERE: the comment defending the one-source-line offence phrase cited selftest 440 grepping this file and counting the copies. **Selftest 440 no longer exists** -- #927 deleted it under the harness-independence rule. Nothing greps this source for the phrase today, so that form is now a convention rather than a guarded property; what is still load-bearing is the RUNTIME string the pytest arms match, which a split f-string would not change at all. The comment says that instead. Gate: harness_selftest 588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED driver-free job 9 files, 161 passed, psycopg absent from the venv full corpus 241 passed with a cluster on pg18a Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

…th file arrived Rebased onto main (d05e3c3). Deleting `test/selftest/370` turned out to touch three artifacts beyond the part itself, and the rebase surfaced a new item of debt that the inventory arm was built to catch. Artifacts the deletion reaches: - `test/selftest/parts.manifest` still named 370. commandprompt#925's arm "every name in the manifest is a part on disk, so a deletion reddens" caught it, which is the arm doing precisely its job on the first deletion after it landed. - `test/check_ledger.tsv` held 12 rows for 370, all `never`. Removed. - `test/check_ledger_budget.txt`'s census `checks_never_observed_red` goes 826 -> 814. DERIVED from the ledger after the removal rather than typed, because that file says of itself: "it is not a ceiling; it is a measurement that must be true". `suites_not_covered` stays at 250 — deleting a part removes no suite. The 12 ledger rows are also the evidence for the claim made when 370 was deleted. They include "a neutered present arm is caught" and "premise: plan_marker's body was actually cut out of the file": 370 cut the Python body out and re-grepped its own detector. Every arm was a text pin, which is what commandprompt#927's precedent is about. A fourth coupled file arrived with commandprompt#925: `test_mutation_ledger.py` runs `run_all_versions.sh --list-suites`. The set-equality assertion in `test_harness_deps.py` reddened on the rebase naming a file the declaration did not have — the direction the inventory was made a mechanism for, working on something nobody wrote it for. It is now declared, with its mechanism, and CONTEXT.md records both the file and the fact that the arm is what found it. Same mechanism as `test_suite_accounting.py`, so the two are one item of debt twice and should move together. Verified on pg18a: selftest 815 checks, 0 FAIL; driver-free subset 196 passed; the full corpus green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

…more of them `test/selftest/360` pinned the SHAPE of `test/pytest/pgc_vacuity.py` with eleven greps: the unrunnable field is written, something reads it, the read reaches `session.exitstatus`, the override is conditional. A shell arm asserting a grep matches cannot prove a python arm is caught -- commandprompt#927's precedent -- and the behaviour is pinned where it can be observed, by four arms in `test_layer.py` that run pytest inside pytest and assert on the inner run's exit status. MEASURED BEFORE DELETING ANYTHING. With `session.exitstatus = EXIT_INCOMPLETE` made unreachable in `pgc_vacuity.py`, the defect exactly as it shipped, those four arms go from `4 passed` to `2 failed, 2 passed`. Two rather than four is correct, not partial: the other two assert exit 1 for a run with a real failure and exit 0 for a run with nothing unrunnable, and neither outcome moves. Restored and compared byte-for-byte afterwards. THE PART'S OWN JUSTIFICATION WAS STALE. It said those arms "need pytest, psycopg and a virtualenv; CI installs none of them". CI has a `pytest-guards` job that installs pytest pinned from `requirements-test.txt`, asserts psycopg is absent, and runs the database-free file list, which contains `test_layer.py`. WHAT IS LEFT IS THE PERMITTED CROSS-REFERENCE, and there are three of them where the part checked one. The INCOMPLETE exit code was checked; the closed list of unrunnable reasons and the one-line shape an unrunnable check prints were not, though both are written down twice in two languages exactly as the exit code is. All three are now parsed out of both files -- never restated here, which would test this file against itself -- and each has a drifted fixture so the comparison can fail. Two instrument defects of mine, both caught by the new arms' own premises: - The shape parse took the first line matching `UNRUN `, which in `pgc_vacuity.py` is the docstring that spells the shape out for a reader. Requiring a quote before the marker selects the quoted string in both languages and excludes the prose, which opens with a backtick. - The drifted-shape fixture wrote ONE space where the real shape has two, so the parse found nothing and the comparison was empty-against-real. That "differs", for the wrong reason. Ledger: 360's rows regenerated with `pgc_ledger.py merge` from a real pg18a run rather than hand-edited, 17 rows out and 16 in, and `checks_never_observed_red` re-derived to 813. One check name was duplicated inside the part and is renamed, so one ledger row no longer covers two checks. `suites_not_covered` stays at 250: rewriting a part's checks removes no suite. Verified on pg18a: selftest 814 checks, 0 FAIL. `pgc_ledger.py gate` with the registered list and `--against auto`: rc=0, `new this run=0`, ceiling 250. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

OffgridwithJD and others added 2 commits September 10, 2026 13:42

This was referenced Sep 10, 2026

test/pytest: a failed query is not a comparison (#432) #930

Merged

test/pytest: a guard in a teardown cannot fail the test it guards (#432) #931

Merged

jdatcmd requested changes Sep 10, 2026

View reviewed changes

OffgridwithJD mentioned this pull request Sep 10, 2026

docs: the two test harnesses are parallel, not coupled #932

Merged

jdatcmd approved these changes Sep 10, 2026

View reviewed changes

jdatcmd merged commit eb7d978 into commandprompt:main Sep 10, 2026
12 checks passed

OffgridwithJD mentioned this pull request Sep 10, 2026

test: the harness guards run in the gate, without a database (#432) #921

Merged

OffgridwithJD mentioned this pull request Sep 10, 2026

test: count a check and record it in one operation (#917) #923

Merged

jdatcmd mentioned this pull request Sep 10, 2026

The pytest harness counts assertions but records nothing, so no count can be reconciled #937

Open

OffgridwithJD mentioned this pull request Sep 10, 2026

test/pytest: the raiser has to be the statement under test (#432) #939

Merged

Navigation Menu

Uh oh!

test/pytest: a broad raises must name a SQLSTATE (#432) - #927

test/pytest: a broad raises must name a SQLSTATE (#432)#927
jdatcmd merged 3 commits into
commandprompt:mainfrom
OffgridwithJD:audit/432-raises-must-name-a-sqlstate

OffgridwithJD commented Sep 10, 2026

Uh oh!

jdatcmd left a comment

Uh oh!

OffgridwithJD commented Sep 10, 2026

Uh oh!

jdatcmd left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

OffgridwithJD commented Sep 10, 2026

What the layer now refuses

What it does NOT close, stated rather than implied

Evidence

The merge commit

Relationship to the open stack

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

1. The pin is satisfied by mentioning .sqlstate, with nothing asserted (blocking)

2. All three "a neutered arm is caught" fixtures are unfaithful, and the faithful neutering stays green (blocking)

3. The keyword form escapes both rules (should-fix)

What I could not fault

Uh oh!

OffgridwithJD commented Sep 10, 2026

1. Mentioning .sqlstate satisfied the pin

2. The keyword form escaped both rules

3. The neutering fixtures — and my first fix for them was wrong

Evidence

What I am not claiming

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. The pin is satisfied by mentioning `.sqlstate`, with nothing asserted (blocking)

1. Mentioning `.sqlstate` satisfied the pin